The present invention relates to a method and an associated apparatus for directly sensing both the geometric and photometric visual information in a three-dimensional (3D) scene. Geometric information specifies the three-dimensional geometry or shape of visible surfaces in the scene. Photometric information specifies the brightness and color of visible surfaces in the scene. The method and apparatus are based on a novel mathematical transform named Rao Transform (RT) for accurately modeling the forward image formation process in a camera which is in general a linear shift-variant integral operation. This new transform and its inverse—the Inverse Rao Transform (IRT)—were invented recently by the author of the present invention. IRT facilitates accurately inverting the shift-variant image formation and sensing in a digital camera to recover the 3D scene. While RT provides an exact method for modeling the forward optics in an image forming optical system, IRT provides a direct and exact method for inverse optics or inverting the image formation in an optical system to recover the visual information in a 3D scene. Therefore, the method of the present invention is named the Direct Vision Sensing Method (DVSM) and the apparatus of the present invention is named the Direct Vision Sensor(DVS).
The 3D geometric and photometric information of the scene recovered using the Rao Transform is used in computer vision applications. Also, this 3D scene information is provided as input to a virtual digital camera which computes the digital still image. This same 3D information for a time-varying scene is used by a virtual video camera to compute and produce digital video data. This video data generation method can be used to create animation videos by modifying the photometric and 3D geometric information based on desired 3D motion, deformation, color change, illumination change, etc., and then computing the video data for a camera with desired camera parameters.